Web pages are complicated entities, made up primarily of Hypertext Markup Language (HTML), but often containing other technologies, such as Cascading Style Sheets (CSS), JavaScript, Flash, and many more. Web Pages can be thought of as programs executed by a browser or client, which is capable of executing software code in the above-mentioned languages and technologies. Without a typical user's knowledge, web pages are often generated upon request, created by running dedicated software on the server when a user request is received. Such dedicated software is called a web application, and uses technologies such as J2EE, PHP, ASP.NET and others.
A web page is defined hereafter as software code for example provided or served as a response to a request for a particular and unique URL or web address, or pointer thereto from a client such as HTML, XHTML or different versions thereof; a web page is therefore software code that embodies or defines the web page, i.e. the software code which allows a web client to render or display a page for viewing.
Therefore a web page at a particular address or pointed thereto whether modified or not is considered to be “the web page”. For all intents and purposes, within the context of this document, if the response to a request for a web page is altered or transformed as compared to a previous response to the same request, the transformed web page is considered to be a modified version of the “same” web page rather than a “new” web page.
One implication of the complexity of web pages is that there are many ways to achieve a same goal. Two web pages can look the same and function the same way for a given client, but their actual content may be very different.
Even when different implementations result in the same interface presented to a user, they may differ greatly in many different aspects. For example, one page may render much faster than the other; one page may expose a security flaw, while the other does not. One page can be successfully loaded in multiple different internet clients or browsers, while the other may only work in Internet Explorer. These are but a few of the implications the specific implementations carry.
Changing a web page, especially one that is auto-generated, can be costly. Web applications are embodied in software, and making modifications to them requires development work, subsequent testing and deployment, all of which risk the integrity of the software and require skilled workers' time. Some of these changes require more expertise and time than others. For example, making a web page load faster, fixing a security flaw or making a web page accessible, often require a fair amount of expertise and time.
Note that some changes to web pages are designed and built into the web application. For example, a news site would read the news articles to display from a database; a personalized home page would serve a different structure for different users, and drag-and-drop functionality may only be included in web pages served back to specific browsers able to support it. In the context of this document, changes to the response based on such logic are considered a part of the web page when built into the web application.
To avoid or reduce the cost of making such changes, these changes are sometimes performed by manipulating the web page after it is generated, using a proxy. A proxy is a software application able to intercept and modify incoming and outgoing communication with the web server. A proxy can be implemented in various ways, including the provision of a separate machine that traffic to a web server would go through, or a software proxy deployed as a web-server add-on through which Internet traffic is passed.
Because Internet traffic it is intercepted by a proxy, it can modify the responses that are returned. For example, a proxy can compress the web page and add an HTTP header indicating it did so, such as the Apache mod_deflate add-on. Making the modifications in a proxy is an alternative to modifying the web application, and provides several benefits with respect to cost (it is often lower cost), time to deploy (it can often be up and running more quickly), flexibility (it is more dynamic in nature, easier to add/remove as needed), and field deployment (it can be deployed and configured by people outside the development group, specifically by those who administer the infrastructure of the website).
Proxy-based manipulations of web pages are relatively common. They generally do not modify the page content, but rather the delivery mechanism wrapper—usually a Hypertext Transfer Protocol (HTTP) response. The modifications performed are typically based on manual configuration, stating which changes to apply under which conditions. The proxies rarely attempt to parse nor do they generally have built in intelligence to understand the pages presented.
In the last few years, there have been a few examples of proxies that not only perform the transformation, but also attempt to analyze the page and transform it based on that analysis. The two primary examples are HTML transcoders for mobile browsing and transformation for performance optimization.
HTML Transcoders for mobile browsing attempt to modify web pages to look better on the smaller smartphone screens. They try to extract the primary information and design aspects of the page, and modify them to fit on a page. These transcoders exist both as proxy solutions and client-side solutions. The different proxy solutions modify the page anywhere between the client and the server, while the client-side solutions modify the page just before rendering it, usually running on the mobile device itself. These HTML Transcoders perform the analysis of the web pages in real-time, while the client is awaiting the response.
Performance optimization analysis and transformation tools analyze pages looking for a variety of known performance related impediments, and attempt to obviate or correct them. For example, one optimization technique is to combine all CSS files referenced in a page into one CSS file. If a page referenced five external CSS files, combining them into one would eliminate four requests when loading the page, and the combined file, when encoded using gzip compression, would likely compress more efficiently than compressing the files separately. A proxy solution may attempt to identify the CSS files in a given page, create a combined file, and modify the page to reference that combined CSS file instead. More examples of web page performance optimizations are explained further down the document.
Performing analysis alongside the transformation makes these proxy solutions much more powerful than transforming based only on configuration. For example, these solutions are more maintainable, as they adapt to changes in the web application or its content without (or with less) user involvement. These analysis solutions also usually provide built-in expertise in the analysis process, knowing what aspects of a page to look for and how to transform them into the desirable result. Performance optimization is a good example of an area where many developers do not know how to make their web pages load faster, making a solution that automatically applies performance optimizations to web application appealing.
Proxy based analysis and transformation engines face two conflicting requirements; the first is speed. In order to transform web pages in real-time, the transformation must be applied quickly enough so as to not introduce any substantial delay. This tolerance for any delay is usually measured in milliseconds, and becomes even more challenging when the web application is under a heavy user load. This requirement is even more important for solutions looking to optimize the performance of a web page, as any delay introduced takes away from the solution's value.
The second is deep analysis. As mentioned above, web pages contain many technologies, and properly understanding a web page is a complicated and CPU intensive task. The most common technology manifesting this problem is JavaScript. While parsing HTML can be done quite efficiently, fully understanding what a snippet of JavaScript code does requires considerably more CPU power and more sophisticated algorithms. Some code snippets are thought to be impossible to analyze in a reasonable time, at least based on current research.
These two requirements are in conflict. On one hand, one cannot perform deep analysis in real-time speed. On the other, without deep analysis, only very basic understanding of a page can be achieved, and the resulting transformations are very limited and error prone.
Let us consider an example of a case where deep analysis is required for the optimization mentioned before, which attempts to merge all referenced CSS files on a page into one file.
Web pages often use JavaScript to dynamically add content to the HTML page, using the document.write( ) function, for various reasons. Consider the following web page referred to hereafter as PAGE 1: <html><head><link rel=‘stylesheet’ type=‘text/css’ href=‘/main.css’><script> var menuVer=‘3.0.2’; document.write (‘<link rel=“stylesheet” href=“/menu.’+menuVer+‘.css>’); </script></head><body><!-- document body here --></body></html>.
This web page references two CSS files. The first reference is written clearly into the HTML. The second reference is printed by JavaScript, using a variable holding the current menu version.
Performing only HTML parsing on this page would conclude there is only one CSS file, and not two, and therefore would not perform the merging of CSS files (or perform it without the menu CSS). However, as mentioned before, executing or statically analyzing JavaScript is complex and resource intensive task, and cannot today be done fast enough to meet the real-time speed requirements.
Web pages are becoming more and more complex and the technologies incorporated within them are becoming more dynamic and difficult to understand. Therefore, analyzing web pages is a task that will only become more difficult over time, and this conflict will only worsen.
Today, the attempted solutions to this problem only raise the threshold of what can be analyzed quickly, or revert to manual configuration for areas that cannot be analyzed fast enough.
One very common solution is to use hardware acceleration, building a dedicated appliance that does all or part of the analysis in hardware. This is an effective solution for some types of analysis, but it only slightly increases the types of analysis that can be done in real-time. For example, executing all the JavaScript on a page cannot be done nearly fast enough, even on hardware, for an average page. One drawback of this type of solution is that it is not very flexible and since the hardware is dedicated to do a particular task, varying that task or addition additional functionality can be problematic.
Another solution is to use heuristics to compensate for shortcomings of analysis. For example, looking for document.write( ) calls in JavaScript code using regular expressions can often be done fast enough for real-time. These techniques are much more error prone and far less complete than the full analysis that could be done with more time.
Another common solution is to use manual configuration to compensate for the more shallow analysis. For example, the user could manually specify that the page above contains the two CSS references. This technique has the original problem of being extremely hard to maintain and is not viable for many dynamic websites.
No solution today offers a complete remedy to this problem. They only attempt to stretch the boundaries a little more is by attempting to analyze faster or attempting to make-do with shallow analysis, however, the subject matter disclosed herein addresses this problem by providing a method and system that among other things speeds up the time in which a web page is returned to a client requesting that page.